Method for recording and replaying mouse commands by recording the commands and the identities of elements affected by the commands

ABSTRACT

An application program includes an action processor which receives messages containing user syntactic actions. These actions are translated into semantic commands. The semantic commands are sent to a command processor for execution. 
     The preferred embodiment of the computing system additionally includes an agent engine. The agent engine may be used to perform many functions. It may be used to receive semantic commands from an application, and to record the semantic commands for later playback. It may be used to send semantic commands from a task language file to an application program for execution by the command processor. It may be used to intercept semantic commands sent from action processor to the command processor. After the command is intercepted, the agent engine may be used to allow the semantic command to be executed, to prevent the semantic command from being executed.

BACKGROUND

The present invention relates to the use of an agent to compile, record,playback and monitor commands used by programs running on a computer.

In many application programs there is a facility for recordingkeystrokes made by a user in interacting with the application program.These keystrokes, stored in a macro file, may be later played back. Thisuse of playback using a macro can allow a user to simply re-execute acomplicated set of commands. Additionally, the user can simplify down tothe running of a macro an often repeated task.

Typically, this type of use of macros has been utilized on a syntaxlevel. What is meant herein by "syntax level" is the action a usermakes, such as keystrokes or movements of a mouse, in order to interactwith an application. For instance, macro files created for laterplayback, typically store a series of keystrokes. An applicationexecuting a macro merely replays the stored keystrokes, and executesthem as if a user were typing the keystrokes on the keyboard.

To simplify the creation of macro files, an application often has a"record" mode which allows a user to interact with the applicationprogram to perform a task. The keystrokes the user uses in performingthe task are recorded in a macro file. The macro file then may be playedback whenever it is desired to repeat the task.

Although storing keystrokes in macro files for playback is a usefulpractice, it is inadequate in many respects. For example, currentschemes for storing keystrokes in macro files are application dependent.They are implemented by a particular application which has its own setof standard rules. Further, such schemes operate syntactically,requiring a user to understand the syntax of a particular application inorder to create a macro file which will operate correctly on thatapplication. Additionally, there is no feedback inherent in the systemto account for any differences in the location or state of objectsbetween the time the keystrokes are recorded and the time the keystrokesare played back. Furthermore, there is typically no way to create macrofiles which when played back operate outside the particular applicationby which the macro file is created.

SUMMARY OF THE INVENTION

In accordance with the preferred embodiments of the present invention acomputing system is presented which includes a plurality ofapplications. Each application program includes an action processorwhich receives messages containing user syntactic actions. These actionsare translated into semantic commands. The semantic commands are sent toa command processor for execution.

The preferred embodiment of the computing system additionally includesan agent engine. The agent engine may be used to perform many functions.It may be used to receive semantic commands from an application, and torecord the semantic commands for later playback. It may be used to sendsemantic commands from a task language file to an application programfor execution by the command processor. It may be used to interceptsemantic commands sent from action processor to the command processor.After the command is intercepted, the agent engine may be used to allowthe semantic command to be executed or to prevent the semantic commandfrom being executed. The ability to intercept semantic commands isespecially useful in computer based training.

The present invention allows greta versatility in the ability of a userto interact with an application. The user may record, playback andmonitor actions performed by an application at the semantic commandlevel, rather than the user syntactic level. This and other advantagesof the present invention are evident from the description of thepreferred embodiment below.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram which shows the interaction between anapplication, an agent environment and a help environment.

FIG. 2 is a block diagram which shows how a task language file isgenerated and executed in accordance with the preferred embodiment ofthe present invention.

FIG. 3 is a block diagram of the application shown in FIG. 1 inaccordance with a preferred embodiment of the present invention.

FIG. 4 is a block diagram showing data flow through the applicationshown in FIG. 1 in accordance with a preferred embodiment of the presentinvention.

FIG. 5 is a diagram of a compiler in accordance with a preferredembodiment of the present invention.

FIG. 6 shows a computer, monitor, keyboard and mouse in accordance withthe preferred embodiment of the present invention.

FIG. 7 shows a top view of the mouse shown in FIG. 6.

FIGS. 8, 9, 10, 11, 12, 13, 14, 15, 16, 17 and 18 show how the displayon the monitor shown in FIG. 6 appears in a user session during whichuser actions are recorded and played back in accordance with thepreferred embodiment of the present invention.

FIG. 19 shows data flow within the compiler shown in FIG. 5.

DESCRIPTION OF THE PREFERRED EMBODIMENT

FIG. 1 is a block diagram of a computing system in accordance with apreferred embodiment of the present invention. A user 111 communicateswith the computing system through a software environment 112. Softwareenvironment 112 may be, for instance, Microsoft Windows, a program soldby Microsoft Corporation, having a business address at 16011 NE 36thWay, Redmond, Wash. 98073-9717. Software environment 112 interacts withan application 100. Messages containing information describing useractions are sent to application 100 by software environment 112. In thepreferred embodiment the messages containing user actions are standardmessages sent by Microsoft Windows. Application 100 includes an actionprocessor 101 which converts syntactic user actions to a single semanticcommand. For example, action processor 101 observes the clicks andmovements of a mouse used by a user, and waits until a syntacticallymeaningful command has been generated. Action processor 101 is able tosyntactically interpret the many ways a user can build a particularcommand. In addition to syntactic user actions, action processor 101also processes other messages from which come to application 100. Somemessages will result in a semantic command being generated; others willbe dealt with entirely by action processor 101.

Application 100 also includes a command processor 102 which executessemantic commands. Command processor 102 receives semantic commands ininternal form (internal form is discussed more fully below) and returnsan error if a command cannot be executed.

Application 100 and software environment 112 interact with helpenvironment 119 at the level of the interface between softwareenvironment 112 and application 100. Help environment 119 includes ahelp application 103, which utilizes a help text 104. Help environment119 also includes help tools 105 which are used to generate help text104.

Software environment 112 also interacts with an agent environment 118.Agent environment 118 includes an agent task 107 and an agent engine108.

Agent engine 108 interacts with application 100 at five differentconceptual categories, in order to perform five functions. Agent engine108 interacts with action processor 101 through a data channel 113 forthe purpose of interrogation. Agent engine 108 interacts between actionprocessor 101 and command processor 102 through a data channel 114 forthe purpose of monitoring the activities of application 100. Agentengine 108 interacts with command processor 102 through a data channel115 for the purpose of having commands executed by application 100.Agent engine 108 interacts with command processor 102 through a datachannel 116 for the purpose of handling errors in the processing of acommand within application 100. Agent engine 108 interacts with commandprocessor 102 through a data channel 117 for the purpose of recordingexecution of application 100 and receiving notification of thecompletion of a command.

In the preferred embodiment of the present invention, commands may berepresented in four ways, (1) in task language form, stored as keywordsand parameters, (2) in pcode form, which are binary codes in externalform with additional header interpreted by agent 108; (3) in externalform, which are binary data understood by application 100 and which arepassed between agent 108 and application 100; and (4) in internal form,as binary commands which are executed within application 100. The fourways of representing commands are further described in Appendix Aattached hereto.

FIG. 2 shows a block diagram of how the overall agent system functions.A task language file 131 is a file containing task language. Tasklanguage is the text form of commands that describe an application'sfunctionality. Task language is comprised of class dependent commandsand class independent commands. Class dependent commands are commandswhich are to be performed by an application. In FIG. 2, just oneapplication, application 100 is shown; however, agent 108 may interactwith many applications.

In the preferred embodiment of the present invention, data files to beoperated on by applications are referenced by the use of objects. Eachobject contains a reference to a data file and a reference to anapplication. Those objects which refer to the same application are saidto be members of the same class. Each application executes a differentset of commands. Class dependent commands therefore differ fromapplication to application.

Agent 108 executes class independent commands which are commandsunderstood by agent 108. Class independent commands are executed byagent 108, not by an application.

Task language file 131 is used by a class independent parser 122 toprepare a pcode file 121. In preparing pcode file 121, independentparser 122 calls class dependent parsers 123, 124 and etc. As will befurther described below, a class dependent parser is a parser whichgenerates class dependent commands which are encapsulated in pcode form.Agent 108 extracts the commands in their external form from the pcodeform and forwards these commands to the appropriate application. A classfield within the pcode indicates which application is to receive aparticular class dependent command. Class independent parser 122 is aparser which generates pcodes which are executed by agent 108.

Task language file 131 may be prepared by user 111 with an agent taskeditor 132. Alternately, task language file may be prepared by use of aclass independent recorder 125 which utilizes class dependent recorders126, 127 and etc. Generally, a recorder records the commands ofapplications for later playback. When the computing system is in recordmode, agent task editor 132 receives input from applications, such asshown application 100, which detail what actions agent engine 108 andthe applications take. Applications communicate to agent task editor 132through an application program interface (API) 130. Agent task editor132, forwards data to class independent recorder 125 when the computingsystem is in record mode, and to task language file 131 when agent taskeditor is being used by user 111.

Class independent recorder 125 receives the information and builds tasklanguage file 131. When class independent recorder 125 detects thatagent task editor 132 is forwarding information about an action taken byan application, class independent recorder calls the class dependentrecorder for that application, which then generates the task languageform for that action. Class independent recorder 108 generates the tasklanguage form for actions taken by agent engine.

When executing pcode 121, agent engine 108 reads each pcode command anddetermines whether the pcode command contains a class independentcommand to be executed by agent 108, or a class dependent command to beexecuted by an application. If the pcode command contains a classindependent command, agent 108 executes the command. If the pcodecommand contains a class dependent command, agent 108 determines by thepcode command the application which is to receive the command. Agent 108then extracts a class dependent command in external form, embeddedwithin the pcode. This class dependent command is then sent to theapplication. For instance, if the class dependent command is forapplication 100, the class dependent command is sent to application 100.Within application 100 a translate to internal processor 128 is used totranslate the class dependent command--sent in external form--to thecommand's internal form.

In the interactions between agent engine 108 and application 100, API130 is used. API 130 is a set of functions and messages for accessingagent engine 108 and other facilities.

When the system is in record mode, translate to internal processor 128translates commands from agent engine 108 and feeds them to commandprocessor 102 through a command interface component 146 shown in FIG. 3.A translate to external processor 129 receives commands in internal formthat have been executed by command processor 102. The commands arereceived through return interface component 147, shown in FIG. 3.Translate to external processor 129 translates the commands in internalform to commands in external form. The commands in external form arethen transferred through API 130 to task editor 132.

FIG. 3 shows in more detail the architecture of application 100 in thepreferred embodiment of the present invention. Application 100 includesa user action interface component 145 which interacts with softwareenvironment 112 and command interface component 146 which communicateswith both action processor 101 and command processor 102. As shown bothaction processor 101 and command processor 102 access application data144. A return interface component 147 is responsive to command processor102 and returns control back to software environment 112. Translate toexternal processor 129 is shown to interact with return interfacecomponent 147. Return interface component 147 is only called whenapplication 100 is in playback mode or record mode. These modes are morefully described below. Return interface component 147 indicates to agentengine 108 that a command has been executed by application 100 andapplication 100 is ready for the next command.

Also included in application 100 are a modal dialog box processor 148and an error dialog box component 149. Both these interact with softwareenvironment 112 to control the display of dialog boxes which communicatewith a user 111.

Some applications are able to operate in more than one window at a time.When this is done a modeless user action interface component, a modelessaction processor, and a modeless command interface component is addedfor each window more than one, in which an application operates. Forexample, in application 100 is shown a modeless user action interfacecomponent 141, a modeless action processor 142 and a modeless commandinterface component 143.

FIG. 4 shows data flow within application 100. Messages to application100 are received by user action interface component 145. For certaintypes of messages--e.g., messages from help application 103--user actioninterface 145 causes application 100 to return immediately. Otherwisethe message is forwarded to a playback message test component 150.

If the message is for playback of commands which have been producedeither by recording or parsing, the message is sent to translate tointernal processor 128 which translates a command within the messagefrom external form to internal form. The command is then forwarded tocommand interface component 146.

If the message is not a playback message the message is sent to actionprocessor 101 to, for example, syntactically interpret a user's actionwhich causes the generation of the message. If there is no semanticcommand generated by action processor 101, or produced by internalprocessor 128 playback message test component 150 causes application 100to return. If there is a semantic command generated the command isforwarded to command interface component 146.

If agent 108 is monitoring execution of commands by application 100,command interface component 146 sends any data received to translate toexternal processor 129 which translates commands to external form andtransfers the commands to agent 108. Command interface component alsoforwards data to a modal dialog box test component 152.

If the forwarded data contains a request for a dialog box, modal dialogbox test component 152 sends the data to modal dialog box processor 148for processing. Otherwise modal dialog box test component 152 sends thedata to command test component 151.

If the data contains a command, command test component 151 sends thecommand to command processor 102 for execution. Command test component151 sends the data to return interface component 147.

If agent 108 is recording commands, return interface component 147 sendsthe data to translate to external processor 129 for translation toexternal form and transfer to agent 108 via return interface component147. Return interface component returns until the next message isreceived.

The following discussion sets out how actions may be recorded and playedback according to the preferred embodiment of the present invention.

In FIG. 8 an application "NewWave Office" is running in a window 205 asshown. Within window 205 is shown a object "Joe" represented by icon201, a folder "Bill" represented by an icon 206, and a folder "Sam"represented by an icon 202. Object "Joe" contains a reference to a textfile and a reference to an application which operates on the text file.Folder "Sam" has been opened; therefore, icon 202 is shaded and a window204 shows the contents of Folder "Sam". Within folder "Sam" is a folder"Fred" represented by an icon 203. A cursor 200 is controlled by a mouse20 or a keyboard 19, as shown in FIG. 6.

FIG. 6 also shows a computer 18 and a monitor 14 on which window 205 isshown. FIG. 7 shows mouse 20 to include a button 27 and a button 28.

Object "Joe" may be placed in folder "Bill" by using mouse 20 to placecursor 200 over object "Joe", depressing button 27, moving cursor 200over folder "Bill" and releasing button 27. Similarly, object "Joe" maybe placed within folder "Sam" by using mouse 20 to place cursor 20 overobject "Joe", depressing button 27, moving cursor 200 within window 204and releasing button 27. Finally, object "Joe" may be placed in folder"Fred" by using mouse 20 to place cursor 20 over object "Joe",depressing button 27, moving cursor 200 over folder "Fred" and releasingbutton 27.

Placement of object "Joe" in folder "Fred", within folder "Sam" or infolder "Bill" may be recorded as will now be described. Each time a usermoves mouse 20, a message containing a syntactic user action is receivedby user action interface component 145, and relayed to action processor101 through playback message test component 150. Based on thesesyntactic user actions, action processor 101 generates a semanticcommand which is executed by command processor 102.

The following describes the recording of the placement of object "Joe"in folder "Bill". In FIG. 8, window 205 is active. Cursor 200 may bemoved about freely in window 205. When user moves mouse 20, syntacticuser actions are sent to action processor 101 as described above. Actionprocessor 101 keeps track of the coordinate location of cursor 200. Whenbutton 27 is depressed, action processor 101 checks to see what existsat the present coordinate location of cursor 200. If cursor 200 isplaced over object "Joe" when button 27 is depressed, action processor101 discovers that object "Joe" is at the location of cursor 200. Atthis time action processor 101 generates a semantic command "SelectDocument `Joe`". The semantic command is passed through playback messagetest component 150, through command interface component 146 throughmodal dialog box test component 152 through command test component 151to command processor 102, which performs the semantic command. Thesemantic command is also received by Return Interface Component 147 ansent to translate to external processor 129. Translate to externalprocessor puts the command in external form and sends it to classindependent recorder 125 and thus to class dependent recorder 126 whichrecords the command in task language form in a task language file.

As mouse 20 is moved syntactic user actions continue to be sent toaction processor 101. Action processor continues to keep track of thecoordinate location of cursor 200. In FIG. 9, cursor 200 is shown to bemoving a "phantom" of object "Joe". In FIG. 10, cursor 200 is shown tobe placed over folder "Bill".

When button 27 is released, action processor 101 generates a semanticcommand "MOVE₋₋ TO Folder `Bill`". The semantic command is passed tocommand processor 102, which causes the previously selected object "Joe"to be transferred to folder "Bill". FIG. 11, shows the completedtransfer, object "Joe" is in folder "Bill". Translate to externalprocessor 129 puts the command in external form and sends it to classindependent recorder 125 and thus to class dependent recorder 126 whichrecords the command in a task language file. When folder "Bill" isopened, as shown in FIG. 12, object "Joe" may be seen.

In this case translate to external processor 129 did not have to getadditional information about object "Joe" or folder "Bill", becauseapplication "NewWave Office" has within itself information thatindicates that object "Joe" and folder "Bill" are on its desktop.Additionally, application 100 "NewWave Office" knows that folder "Bill"is closed.

Recording of the placement of object "Joe" within folder "Sam" issimilar to the above. In FIG. 8, window 205 is active. Cursor 200 may bemoved about freely in window 205. When button 27 is depressed, actionprocessor 101 checks to see what exists at the present coordinatelocation of cursor 200. If cursor 200 is placed over object "Joe" whenbutton 27 is depressed, action processor 101 discovers that object "Joe"is at the location of cursor 200. At this time action processor 101generates a semantic command "Select Document`Joe`". The semanticcommand is passed through playback message test component 150, throughcommand interface component 146 through modal dialog box test component152 through command test component 151 to command processor 102, whichperforms the semantic command. The semantic command is also received byReturn Interface Component 147 and sent to translate to externalprocessor 129. Translate to external processor puts the command inexternal form and sends it to class independent recorder 125 and thus toclass dependent recorder 126 which records the command in a tasklanguage file.

As mouse 20 is moved syntactic user actions continue to be sent toaction processor 101. Action processor continues to keep track of thecoordinate location of cursor 200. In FIG. 13, cursor 200 is shown to beplaced within window 204. When button 27 is released, action processor101 generates a MOVE₋₋ TO Folder "Sam" command. The semantic command ispassed to command processor 102, which causes the previously selectedobject "Joe" to be transferred to folder "Bill". The semantic command isalso received by return interface component 147 and sent to translate toexternal processor 129. Translate to external processor 129 sends an"API₋₋ INTERROGATE₋₋ MSG". The function of the message is "API₋₋ WHO₋₋ARE₋₋ YOU₋₋ FN". As a result of this message, translate to externalprocessor 129 gets returned data indicating that an open window forfolder "Sam" is at the location of cursor 200. Translate to externalprocessor 129 sends another "API₋₋ INTERROGATE₋₋ MSG". The function ofthe message is again "API.sub. -- WHATS₋₋ INSERTABLE₋₋ AT₋₋ FN". Sincethere there is nothing within window 204 at the location of cursor 200,no additional entity is identified. For a further description of API₋₋INTERROGATE₋₋ MSG see Appendix C.

Translate to external processor puts the command in external form andsends it to class independent recorder 125 and thus to class dependentrecorder 126, and the command is recorded in task language file 131.FIG. 14 shows the result of the completed transfer: object "Joe" iswithin window 204.

Similarly object "Joe" may be transferred to folder "Fred". In FIG. 15,cursor 200 is shown to be placed over folder "Fred" within window 204.When button 27 is released, action processor 101 generates a semanticcommand "MOVE₋₋ TO Folder `Fred` WITHIN Folder `Sam`". The semanticcommand is passed to command processor 102, which causes the previouslyselected object "Joe" to be transferred to folder "Fred" within Folder"Sam". The semantic command is also received by return interfacecomponent 147 and sent to translate to external processor 129.

Translate to external processor 129 puts the command in external form inthe following manner Translate to external processor 129 sends an "API₋₋INTERROGATE₋₋ MSG". The function of the message is "API₋₋ WHATS₋₋INSERTABLE₋₋ AT₋₋ FN". As a result of this message, translate toexternal processor 129 receives a return message indicating that folder"Fred" is at the location of cursor 200. Translate to external processorsends another "API₋₋ INTERROGATE₋₋ MSG". The function of the message is"API₋₋ WHO₋₋ ARE₋₋ YOU₋₋ FN". As a result of this message, translate toexternal processor 129 receives return data indicating that folder "Sam"is at the location of cursor 200.

At this time translate to external processor is able to send the commandin external form through API 130 to class independent recorder 125 andthus to class dependent recorder 126. Class dependent recorder 126records the external command in task language file 131. FIG. 16, showsthe completed transfer, object "Joe" is in folder "Fred". When folder"Fred" is opened, as shown in FIG. 17, object "Joe" may be seen.

Once in a task language file, the commands which transferred object"Joe" to folder "Fred", may be played back. For instance, suppose window205 appears as in FIG. 18. Since window 204, object text "Joe" andfolder "Fred" are all in different locations within window 205, a mereplayback of syntactic user actions would not result in object "Joe"being placed within folder "Fred". However, what was recorded was notsyntactic user actions but rather semantic commands; therefore, playbackof the semantic commands will cause object "Joe" to be placed withinFolder "Fred".

Specifically, suppose a task language file contained the followingcommands:

FOCUS on Desktop "NewWave Office"

SELECT Document "Joe"

MOVE₋₋ TO Folder "Fred" WITHIN Folder "Sam".

The first command--FOCUS on Desktop "NewWave Office"--is a classindependent command and, once compiled by a task language compiler 120shown in FIG. 5, may be executed by agent 108. As will be furtherdescribed below, the FOCUS command places the focus on the application"NewWave Office". This means that the task language commands are, ifpossible, to be treated as class dependent commands and sent toapplication "NewWave Office" for execution. For simplicity ofdiscussion, the application "NewWave Office" is taken to be application100.

The second and third commands --SELECT Document "Joe"--and --MOVE₋₋ TOFolder "Fred" WITHIN Folder "Sam"-- are class dependent commands. Theseclass dependent commands, once compiled by task language compiler 120into pcode form, are received by agent engine 108. Agent engine extractsthe class dependent commands in external form from the pcode form andsends the class dependent commands to application 100. User actioninterface component 145 of application 100 receives a message containingthe external command and forwards the message to playback message testcomponent 150. Playback message test component 150 ships the command totranslate to internal processor 128. Translate to internal processor 128translates the command from external form to internal form and returnsthe command in internal form to playback test component 150. The commandin internal form is then sent through command interface component 146,through modal dialog box test component 152 through command testcomponent 151 to command processor 102. Command processor 102 executesthe command.

Agent 108 executes the command "FOCUS on Desktop `NewWave Office`", byactivating window 205. The position of cursor 200 is now determined withrespect to the coordinates of window 205.

When command processor 102 receives the command "SELECT Document `Joe`",command processor 102 causes object "Joe" to be selected. Since object"Joe" is within window 205 no additional interrogation is necessary.

When constructing the internal command form for the command "MOVE₋₋ TOFolder `Fred` WITHIN Folder `Sam`", translate to internal processor 128sends an "API₋₋ INTERROGATE₋₋ MSG" to each open window. The function ofthis message is "API₋₋ WHO₋₋ ARE₋₋ YOU FN".

When the window for Folder "Sam" receives this message, it responds with"Folder `Sam`". Translate to internal processor 128 sends another"API₋₋INTERROGATE₋₋ MSG". The function of this message is "API₋₋ WHERE₋₋ IS₋₋FN". Folder "Fred" is included as a parameter. The message is forwardedto folder "Sam" which returns data indicating the coordinates of folder"Fred" within window 204. Translate to internal processor 128 thengenerates the internal form of the command MOVE₋₋ TO `Fred` WITHINFolder "Sam". Command processor 120 receives the command and transfersobject "Joe" to folder "Fred".

Task language file 121 may be generated by compiled code written by auser, as well as by recording. In FIG. 5, data flow through a tasklanguage compiler 120 is shown. A task language file 131 includescommands written by a user. In the preferred embodiment of the presentinvention, the task language is written in accordance with the AgentTask Language Guidelines included as Appendix B to this Specification.

Task language compiler 120 is a two pass compiler. In the first pass theroutines used include an input stream processor 164, an expressionparser 166, a class independent parser 122, a save file buffer 171,second pass routines 174, and class dependent parsers, of which areshown class dependent parser 123, a class dependent parser 167 and aclass dependent parser 168. As a result of the first pass a temporaryfile 176 is created.

Class independent parser 122 parses the class independent task languagecommands listed in Appendix B. Each application which runs on the systemalso has special commands which it executes. For each application,therefore, a separate class dependent parser is developed. This parseris able to parse commands to be executed by the application for which itis developed. Class dependent parsers may be added to or deleted fromtask language compiler 120 as applications are added to or deleted fromthe system.

When compiling begins, class independent parser 122 requests a tokenfrom input stream processor 164. Input stream processor 164 scans tasklanguage file 131 and produces the token. Class independent parser 122then does one of several things. Class independent parser 122 maygenerate pcode to be sent to save file buffer 171. If class independentparser 122 expects the next token to be an expression, class independentparser 1232 will call routine MakeExpression () which calls expressionparser 166. Expressions parser 166 requests tokens from input streamprocessor 164 until the expression is complete. Expression parser 166then generates pcode to be sent to file buffer 171 and then to be savedin temporary file 176. Additionally, expression parser 166 generates anexpression token which is returned to input stream processor 164. Inputstream processor 164 delivers this expression to independent parser 122when it is requested by independent parser 122.

As a result of a FOCUS command, a particular class dependent parser willhave priority. Therefore, in its parsing loop, class independent scanner122a will call the class dependent parser for the application whichcurrently has the focus. The class dependent parser will request tokensfrom input stream processor 164 until it has received a class dependentcommand which the semantic routines called by class dependent parserconvert to external command form, or until the class dependent parserdetermines that it cannot parse the expressions that it has received. Ifthe class dependent parser encounters an expression, it may invokeexpression parser 166 using the call MakeExpression (). If the classdependent parser is unable to parse the tokens it receives, the classdependent parser returns an error and the class independent parser willattempt to parse the tokens.

A FOCUS OFF command will result in independent parser 122 immediatelyparsing all commands without sending them to a dependent parser. When astring of class independent commands are being parsed, this can avoidthe needless running of dependent parser software, thus saving computingtime required to compile the task language.

In FIG. 19 is shown data flow between independent parser 122 anddependent parsers of which dependent parser 123 and dependent parser 124are shown. In order to focus the discussion on the relationship betweenparsers, calls to expression parser 166 by scanner 122a are not takeninto account in the discussion of FIG. 19.

When independent parser 122 is ready for a token, independent parser 122calls a scanner routine 122a. Scanner 122a checks if there is a focus onan application. If there is not a focus on an application, scanner 122acalls input stream processor 164 which returns to scanner 122a a token.Scanner 122a returns the token to independent parser 122a.

If there is a focus on an application, the dependent parser for theapplication has precedence and is called. For instance, when focus is onthe application for parser 123, parser 123 calls scanner 122a through adependent scanner 123a. Scanner 122a checks its state and determinesthat it is being called by a dependent parser, so it does norrecursively call another dependent parser. Scanner 122a calls inputstream processor 164 which returns to scanner 122a a token. Scanner 122areturns the token to dependent parser 123 through dependent scanner123a. Although the present implementation of the present inventionincludes dependent scanner 123a, in other implementations dependentscanner 123a may be eliminated and parser 123 may call scanner 122adirectly.

Dependent parser 123 will continue to request tokens through dependentscanner 123a as long is dependent parser 123 is able to parse the tokensit receives. With these tokens dependent parser will call semanticroutines which will generate class dependent external commands embeddedin pcode. When dependent parser 123 is unable to parse a token itreceives, dependent parser will return to scanner 122a an error. Scanner122a then calls input stream processor 164 and receives from inputstream processor 164 the token which dependent parser 123 was unable toparse. This token is returned to independent parser 122. Independentparser 122 parses the token and calls semantic routines to generatepcode for execution by agent 108. The next time independent parser 122requests a token from scanner 122a, scanner 122a will again calldependent parser 123 until there is a FOCUS OFF command or until thereis a focus on another application.

When the focus is on the application for dependent parser 124, scanner122a will call dependent parser 124. Dependent parser 124 calls adependent scanner 124a and operates similarly to dependent parser 123.

Save file buffer 171, shown in FIG. 5, receives pcode from classindependent parser 122 an from expression parser 166, and receivesexternal command forms embedded in pcode from class dependent parsers.Save file buffer 171 stores this information in a temporary file 176.Second pass routines 174 takes the pcode and external command formsstored in temporary file 176 and performs housekeeping, e.g., fixesaddresses etc., in order to generate task language file 121.

Appendix A contains an Introduction to API 130 (Programmer's GuideChapter 4).

Appendix B contains guidelines for developing agent task language (AgentTask Language Guidelines).

Appendix C contains a description of Task Language Internals.

Appendix D contains description of API₋₋ INTERROGATE₋₋ MSG. ##SPC1##

We claim:
 1. In a computing system which includes a viewer screen and auser interface which enables a user to select and move images displayedon the viewing screen, a computer implemented method for recording in adata file user commands for later playback, the recording of usercommands requiring syntactic analysis to determine an identity of anentity, the user commands being made by the user via selection andmovement of images on the viewing screen and the user commands beingexecutable by a first application process, the computer implementedmethod comprising the steps, performed by the computing system, of:(a)translating, by the fits application process, selection and movement ofimages on the viewing screen into semantic commands, the translationincluding performance of syntactic analysis of the selection andmovement of images; (b) concurrent with step (a) when syntactic analysisof selection and movement of images on the viewing screen indicate anentity on the computing system is to be operated upon by a semanticcommand and the first application process does not know the identity ofthe entity, performing the following substeps(b.1) generating, by thefirst application process, an interrogation message to identify theentity that is to be operated upon, and, (b.2) returning to the firstapplication process, a response message identifying the entity; and, (c)recording the semantic commands translated in step (a) including theidentity of any entity identified in step (b) in the data file.
 2. Acomputer implemented method as in claim 1 wherein step (c) comprises thesubsteps of:(c.1) translating the semantic commands into task languageform; and, (c.2) recording the semantic commands in task language formin the data file.
 3. A computer implemented method as in claim 2 whereinsubstep (c.1) comprises the substeps of:(c.1.a) translating the semanticcommands into an external command form; and, (c.1.b) translating thesemantic commands in external command form into task language form usinga class dependent recorder.
 4. A computer implemented method as in claim1, additionally comprising the following step which is performedconcurrently with step (c):(d) recording in the data file, semanticcommands which are translated, by a second application process, fromselection and movement of images on the viewing screen which occur whenthe user is interacting with the second application process.
 5. Acomputer implemented method as in claim 1, additionally comprising thefollowing step performed concurrently with step (c):(d) recording intask language form in the data file, actions taken by an agent engine.6. In a computing system which includes a viewer screen and a userinterface which enables a user to select and move images displayed onthe viewing screen, semantic commands being generated by selecting andmoving images on the viewing screen, a computer implemented method forplayback of a plurality of stored semantic commands which are executableby an application process, the computer implemented method comprisingthe steps, performed by the computing system, of:(a) reading from a datafile, a first semantic command from the plurality of semantic commands;(b) receiving, by the application process, the first semantic command;(c) when an entity on the computing system, represented by a first imageon the viewing screen, is to be operated upon by the computing systemwhile executing the first semantic command performing the followingsubsteps,(c.1) generating, by the application process, an interrogationmessage to identify the location of the first image on the viewingscreen, and (c.2) returning to the application process a responsemessage identifying the location of the first image on the viewingscreen; and, (d) executing the first semantic command, by theapplication process, while selecting and moving images on the viewingscreen to identify to the user the first semantic command.
 7. A computerimplemented method as in claim 6 additionally comprising the followingstep performed before step (c):(e) translating the semantic command froman external form to an internal form.
 8. A computer implemented methodas in claim 7 wherein the computer implemented method is additionallyfor generation of semantic commands and the computer implemented methodadditionally comprises the following step performed before step (a):(f)recording by the computer system into the data file for later playback,the plurality of semantic commands, the plurality of semantic commandsbeing generated by the user selecting and moving images on the viewingscreen.
 9. A computer implemented method as in claim 8, wherein step (f)comprises the following substeps:(f.1) translating, by the applicationprocess, selection and movement of images on the viewing screen intosemantic commands, the translation including syntactic analysis upon theselection and movement of images; (f.2) concurrent with substep (f.1),when syntactic analysis of selection and movement of images on theviewing screen indicate an entity on the computing system is to beoperated upon by a semantic command and the application process does notknow the identity of the entity, generating, by the application process,an interrogation message to identify the entity that is to be operatedupon; and, (f.3) recording in the data file the semantic commandsincluding the identity of any identified entity.
 10. A computerimplemented method as in claim 9 wherein the substep (f.3) comprises thefollowing substeps:(f.3.a) translating the semantic commands into tasklanguage form; and, (f.3.b) recording the semantic commands in tasklanguage form in the data file.
 11. In a computing system which includesa viewer screen and a user interface which enables a user to select andmove images displayed on the viewing screen, a computer implementedmethod for recording user commands for later playback the recording ofuser commands requiring synmtactic analysis to determine an identity ofan entity, the user commands being made by the user via selection andmovement of images on the viewing screen and the user commands beingexecutable by a first application process, the first application processcontrolling images in a first portion of the viewing screen, thecomputer implemented method comprising the steps, performed by thecomputing system, of:(a) translating, by the first application process,selection and movement of images on the viewing screen into semanticcommands, the translating including performing syntactic analysis uponthe selection and movement of images; (b) concurrent with step (a) whensyntactic analysis of selection and movement of images on the viewingscreen indicate at least part of an operation is performed in a secondportion of the viewing screen controlled by a second applicationprocess, performing the following substeps,(b.1) generating, by thefirst application process, a first interrogation message, sent to thesecond application process, requesting the second application process toidentify itself to the first application process, and (b.2) returning,by the second application process to the first application process, afirst response message which identifies the second application processto the first application process; and, (c) recording, in a data file,the semantic command and the identity of the second application processwhen the second application process is identified in step (b.2).
 12. Acomputer implemented method as in claim 11 additionally comprising thefollowing steps:(d) generating, by the first application process, asecond interrogation message which identifies a specific location withinthe second portion of the viewing screen and requests the secondapplication process to identify any entity of which an image resides atthe specific location; (e) returning, by the second application processto the first application process, a second response message whichidentifies any entity of which an image resides at the specificlocation; and, (f) additionally recording in the data file the identityof the entity identified in the second response message.
 13. In acomputing system which includes a plurality of application processesrunning on the computing system and which includes a viewer screen and auser interface which enables a user to select and move images displayedon the viewing screen, semantic commands being generated by selectingand moving images on the viewing screen, a computer implemented methodfor playback of stored semantic commands which are executable by a firstapplication process, the first application process controlling images ina first portion of the viewing screen, the computer implemented methodcomprising the steps, performed by the computing system, of:(a) when atleast part of an operation is to be performed in a second portion of theviewing screen controlled by a second application process, performingthe following substeps,(a.1) generating, by the first applicationprocess, a first interrogation message asking application processes fromthe plurality of application processes which control portions of theviewing screen to identify themselves, (a.2) transmitting the firstinterrogation message to the application processes, and (a.3) after thetransmitting in step (a.2), returning by each application processcontrolling a portion of the viewing screen, a response messageidentifying itself; and, (b) after step (a), executing the semanticcommand by the first application.
 14. A computer implemented method asin claim 13 additionally comprising the following steps, performedbefore step (b):(c) generating, by the first application process, asecond interrogation message which requests the second applicationprocess to identify the location on the viewing screen of an image of anentity which is operated on by the semantic command; and, (d) returning,by the second application process to the first application process, aresponse message which identifies the location on the viewing screen ofthe image of the entity.
 15. A computer implemented method as in claim14 wherein step (b) includes selecting and moving images on the viewingscreen to identify to the user the semantic command which is beingexecuted.